Using Curvature Information for Fast Stochastic Search Improving Stochastic Search

We present an algorithm for fast stochastic gradient descent that uses a nonlinear adaptive momentum scheme to optimize the late time convergence rate. The algorithm makes eeective use of curvature information, requires only O(n) storage and computation, and delivers convergence rates close to the theoretical optimum. We demonstrate the technique on linear and large nonlinear back-prop networks. Learning algorithms that perform gradient descent on a cost function can be formulated in either stochastic (on-line) or batch form. The stochastic version takes the form ! t+1 = ! t + t G(! t ; x t) (1) where ! t is the current weight estimate, t is the learning rate, G is minus the instantaneous gradient estimate, and x t is the input at time t 1. One obtains the corresponding batch mode learning rule by taking constant and averaging G over all x. Stochastic learning provides several advantages over batch learning. For large datasets the batch average is expensive to compute. Stochastic learning eliminates the averaging. The stochastic update can be regarded as a noisy estimate of the batch update, and this intrinsic noise can reduce the likelihood of becoming trapped in poor local optima 1, 2]. 1 We assume that the inputs are i.i.d. This is achieved by random sampling with replacement from the training data.